{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tracing, Evaluating and Profiling your Agent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Observing a Workflow with Phoenix\n",
    "\n",
    "We can now go through the steps to enable observability in a workflow using Phoenix for tracing and logging.\n",
    "\n",
    "NeMo Agent toolkit provides comprehensive tracing that automatically monitors all registered functions in your workflow, LLM interactions, and any custom functions decorated with @track_function, capturing their inputs, outputs, and execution flow to provide complete visibility into how your agent processes requests. The lightweight `@track_function` decorator can be applied to any Python function to gain execution insights without requiring full function registration—this is particularly valuable when you want to monitor utility functions, data processing steps, or business logic that doesn't need to be a full NAT component. All tracing data flows into a unified observability system that integrates seamlessly with popular monitoring platforms like Phoenix, OpenTelemetry, and LangSmith, enabling real-time monitoring, performance analysis, and debugging of your entire agent workflow from high-level function calls down to individual processing steps."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To enable tracing, update your workflow configuration file to include the telemetry settings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```yaml\n",
    "general:\n",
    "  telemetry:\n",
    "    logging:\n",
    "      console:\n",
    "        _type: console\n",
    "        level: WARN\n",
    "    tracing:\n",
    "      phoenix:\n",
    "        _type: phoenix\n",
    "        endpoint: http://localhost:6006/v1/traces\n",
    "        project: retail_sales_agent\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the following command to start Phoenix server locally:\n",
    "\n",
    "```bash\n",
    "phoenix serve\n",
    "```\n",
    "Phoenix should now be accessible at http://localhost:6006.\n",
    "\n",
    "Run this using the following command and observe the traces at the URL above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import getpass\n",
    "import os\n",
    "\n",
    "if \"NVIDIA_API_KEY\" not in os.environ:\n",
    "    nvidia_api_key = getpass.getpass(\"Enter your NVIDIA API key: \")\n",
    "    os.environ[\"NVIDIA_API_KEY\"] = nvidia_api_key\n",
    "\n",
    "if \"TAVILY_API_KEY\" not in os.environ:\n",
    "    tavily_api_key = getpass.getpass(\"Enter your Tavily API key: \")\n",
    "    os.environ[\"TAVILY_API_KEY\"] = tavily_api_key\n",
    "\n",
    "if \"OPENAI_API_KEY\" not in os.environ:\n",
    "    openai_api_key = getpass.getpass(\"Enter your OpenAI API key: \")\n",
    "    os.environ[\"OPENAI_API_KEY\"] = openai_api_key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nat run --config_file retail_sales_agent/configs/config_tracing.yml --input \"How do laptop sales compare to phone sales?\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluating a Workflow using `nat eval`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please refer to [this documentation](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/evaluate.html) for a detailed guide on evaluating a workflow.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For evaluating this workflow, we create a sample [dataset](./retail_sales_agent/data/eval_data.json)\n",
    "\n",
    "```json\n",
    "[\n",
    "    {   \n",
    "        \"id\": \"1\",\n",
    "        \"question\": \"How do laptop sales compare to phone sales?\",\n",
    "        \"answer\": \"Phone sales are higher than laptop sales in terms of both revenue and units sold. Phones generated a revenue of 561,000 with 1,122 units sold, whereas laptops generated a revenue of 512,000 with 512 units sold.\"\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"2\",\n",
    "        \"question\": \"What is the Ark S12 Ultra tablet and what are its specifications?\",\n",
    "        \"answer\": \"The Ark S12 Ultra Ultra tablet features a 12.9-inch OLED display with a 144Hz refresh rate, HDR10+ dynamic range, and a resolution of 2800 x 1752 pixels. It has a contrast ratio of 1,000,000:1. The device is powered by Qualcomm's Snapdragon 8 Gen 3 SoC, which includes an Adreno 750 GPU and an NPU for on-device AI tasks. It comes with 16GB LPDDR5X RAM and 512GB of storage, with support for NVMe expansion via a proprietary magnetic dock. The tablet has a 11200mAh battery that enables up to 15 hours of typical use and recharges to 80 percent in 45 minutes via 45W USB-C PD. Additionally, it features a 13MP main sensor and a 12MP ultra-wide front camera, microphone arrays with beamforming, Wi-Fi 7, Bluetooth 5.3, and optional LTE/5G with eSIM. The device runs NebulynOS 6.0, based on Android 14L, and supports app sandboxing, multi-user profiles, and remote device management. It also includes the Pluma Stylus 3 with magnetic charging, 4096 pressure levels, and tilt detection, as well as a SnapCover keyboard with a trackpad and programmable shortcut keys.\"\n",
    "    },\n",
    "    {\n",
    "        \"id\": \"3\",\n",
    "        \"question\": \"What were the laptop sales on Feb 16th 2024?\",\n",
    "        \"answer\": \"On February 16th, 2024, the total laptop sales were 13 units, generating a total revenue of $13,000.\"\n",
    "    }\n",
    "]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nat eval --config_file retail_sales_agent/configs/config_evaluation_and_profiling.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `nat eval` command runs the workflow on all the entries in the dataset. The output of these runs is stored in a file named `workflow_output.json` under the `output_dir` specified in the configuration file.\n",
    "\n",
    "Each evaluator provides an average score across all the entries in the dataset. The evaluator output also includes the score for each entry in the dataset along with the reasoning for the score. The score is a floating point number between 0 and 1, where 1 indicates a perfect match between the expected output and the generated output.\n",
    "\n",
    "The output of each evaluator is stored in a separate file under the `output_dir` specified in the configuration file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Profiling a Workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please refer to [this documentation](https://docs.nvidia.com/nemo/agent-toolkit/latest/workflows/profiler.html) for a detailed guide on profiling a workflow.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The profiler can be run through the `nat eval` command and can be configured through the `profiler` section of the workflow configuration file.\n",
    "\n",
    "Please also note the `output_dir` parameter which specifies the directory where the profiler output will be stored. \n",
    "\n",
    "Let us explore the profiler configuration options:\n",
    "\n",
    "- `token_uniqueness_forecast`: Compute the inter-query token uniqueness forecast. This computes the expected number of unique tokens in the next query based on the tokens used in the previous queries.\n",
    "\n",
    "- `workflow_runtime_forecast`: Compute the expected workflow runtime forecast. This computes the expected runtime of the workflow based on the runtime of the previous queries.\n",
    "\n",
    "- `compute_llm_metrics`: Compute inference optimization metrics. This computes workflow-specific metrics for performance analysis (e.g., latency, throughput, etc.).\n",
    "\n",
    "- `csv_exclude_io_text`: Avoid dumping large text into the output CSV. This is helpful to not break the structure of the CSV output.\n",
    "\n",
    "- `prompt_caching_prefixes`: Identify common prompt prefixes. This is helpful for identifying if you have commonly repeated prompts that can be pre-populated in KV caches\n",
    "\n",
    "- `bottleneck_analysis`: Analyze workflow performance measures such as bottlenecks, latency, and concurrency spikes. This can be set to simple_stack for a simpler analysis. Nested stack will provide a more detailed analysis identifying nested bottlenecks like tool calls inside other tools calls.\n",
    "\n",
    "- `concurrency_spike_analysis`: Analyze concurrency spikes. This will identify if there are any spikes in the number of concurrent tool calls. At a spike_threshold of 7, the profiler will identify any spikes where the number of concurrent running functions is greater than or equal to 7. Those are surfaced to the user in a dedicated section of the workflow profiling report."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the profiler for our created workflow using the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!nat eval --config_file retail_sales_agent/configs/config_evaluation_and_profiling.yml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will, based on the above configuration, produce the following files in the output_dir specified in the configuration file:\n",
    "\n",
    "- `all_requests_profiler_traces.json`: This file contains the raw usage statistics collected by the profiler. Includes raw traces of LLM and tool input, runtimes, and other metadata.\n",
    "\n",
    "- `inference_optimization.json`: This file contains the computed workflow-specific metrics. This includes 90%, 95%, and 99% confidence intervals for latency, throughput, and workflow runtime.\n",
    "\n",
    "- `standardized_data_all.csv`: This file contains the standardized usage data including prompt tokens, completion tokens, LLM input, framework, and other metadata.\n",
    "\n",
    "- You’ll also find a JSON file and text report of any advanced or experimental techniques you ran including concurrency analysis, bottleneck analysis, or PrefixSpan."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
